{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FNSb2vhtFOzp"
   },
   "source": [
    "## Understanding multilingual BERT \n",
    "\n",
    "BERT gives the representation for only the English text. Let's suppose we have an input text in a different language, say, French, now how we can use the BERT for obtaining the representation of the French text? Here is where we use an M-BERT. \n",
    "\n",
    "The multilingual BERT shortly known as M-BERT is used for obtaining the representation of text in different languages and not just English. We learned that the BERT model is trained with masked language modeling and next sentence prediction tasks using the English Wikipedia text and the Toronto BookCorpus. Similar to BERT, the M-BERT is also trained with masked language modeling and next sentence prediction tasks but instead of using the Wikipedia text of only English language, M-BERT is trained using the Wikipedia text of 104 different languages. \n",
    "\n",
    "But the question is, the size of the Wikipedia text for some languages would be higher than the other right? Yes! the size of Wikipedia text would be large for high-resource languages like English compared to low-resource languages like Swahili. If we train our model with this dataset then it will lead to the problem of overfitting. To avoid overfitting, we use sampling methods. We apply under sampling for high-resource languages and over-sampling for low-resource languages. \n",
    "\n",
    "Since the M-BERT is trained over Wikipedia text of 104 different languages, it learns the general syntactic structure of different languages. The M-BERT consists of 110K shared WordPiece vocabulary across all the 104 languages. \n",
    "\n",
    "The M-BERT understands the context from different languages without any paired or language aligned training data. It is important to note that we have not trained M-BERT with any cross-lingual objective, it is trained just like how we trained the BERT model. The M-BERT produces a representation that generalizes across multiple languages for downstream tasks. \n",
    "\n",
    "The pre-trained M-BERT model is open-sourced by Google and it can be downloaded from here - https://github.com/google-research/bert/blob/master/multilingual.md. The various configurations of pre-trained M-BERT models provided by Google are given in the following: \n",
    "\n",
    "- BERT-base, Multilingual cased \n",
    "- BERT-base, Multilingual uncased\n",
    "\n",
    "Both of the preceding models consist of 12 encoder layers, 12 attention heads, 768 hidden zie. It consists of a total of 110 million parameters. \n",
    "\n",
    "The pre-trained M-BERT is also compatible with the Hugging Face's transformers library. So, we can use it with the transformers library just like how we use the BERT. Let us see how to use the pre-trained M-BERT model and obtain the sentence representation: \n",
    "\n",
    "First, let's import the necessary modules:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "id": "8WzuLKGYFQpN"
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "!pip install transformers==3.5.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "id": "TzB07ZdoFOz2"
   },
   "outputs": [],
   "source": [
    "from transformers import BertTokenizer, BertModel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JW2XZfxPFOz3"
   },
   "source": [
    "\n",
    "Download and load the pre-trained M-BERT model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 115,
     "referenced_widgets": [
      "d9604103dc9d42b39678896f1315125d",
      "536d42bf13324e6b9aa2e93490c259c9",
      "9993840b4fce420499c97f4a6c97d23f",
      "0c2d4e7abb974c8dad60d7e1de2eee2e",
      "700df478ee7147bdb9a511d57aca78a9",
      "9fa578e43eb44136879cfd442a506a1c",
      "8836a23e70504ba79fd2d605e855cf58",
      "084def079e504107ac492665028927f5",
      "2d36852fa3c84535b644d8eeae96b117",
      "4d36e204d30c4d0cb8f5d8f4545bf003",
      "9c93a8681a1b497bb724f7b2e6dadefe",
      "08ccac7ca81e4a7abfd0c66767590f37",
      "989710fc8a2c4117b3a49397bdcdc2e5",
      "7d8d6ab425534b4eaab46e866c9c65b7",
      "8ae7e3eb93ab422aa477407a4d3d6705",
      "2e8c53f981b0424f9df0f0e1caa0f73d"
     ]
    },
    "id": "hL-85_FhFOz4",
    "outputId": "0bd62d80-f7c9-4cc7-d02c-c1fe54dfc8b3"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d9604103dc9d42b39678896f1315125d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=625.0, style=ProgressStyle(description_…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2d36852fa3c84535b644d8eeae96b117",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=714314041.0, style=ProgressStyle(descri…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "model = BertModel.from_pretrained('bert-base-multilingual-cased')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tHQh9jSFFOz4"
   },
   "source": [
    "\n",
    "\n",
    "Download and load the pre-trained M-BERT model's tokenizer: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 66,
     "referenced_widgets": [
      "aabc82f34c9948bdb8c3a6fcadbd859b",
      "30735bf2ffd74418b40f63a8f66c9432",
      "32c0276d3e714b0fafa26b8e3e63fb70",
      "8d1edfd37dcf4a95aecb93046bb5d28b",
      "1751eccfd3544e6eace561b54ed028a7",
      "e8f61aa83643415aa917d9ecfbb8d15f",
      "05c3b23cc9174177bcee786ce119534c",
      "4b01f8e03f594685a7c24592d9153d63"
     ]
    },
    "id": "avmGU0-tFOz4",
    "outputId": "7e624c51-224a-4719-8611-9ee1b9df8e11"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "aabc82f34c9948bdb8c3a6fcadbd859b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=995526.0, style=ProgressStyle(descripti…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qzWhTy9jFOz5"
   },
   "source": [
    "\n",
    "\n",
    "Define the input sentence. Let us French sentence as an input: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true,
    "id": "g1unht_JFOz5"
   },
   "outputs": [],
   "source": [
    "sentence = \"C'est une si belle journée\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cDcoOMN9FOz5"
   },
   "source": [
    "\n",
    "Tokenize the sentence and get the tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "id": "lOsTFIMVFOz6"
   },
   "outputs": [],
   "source": [
    "inputs = tokenizer(sentence, return_tensors=\"pt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jHjWY21pFOz6"
   },
   "source": [
    "\n",
    "Feed the tokens to the model and get the representation: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "id": "eykNs5nEFOz6"
   },
   "outputs": [],
   "source": [
    "hidden_rep, cls_head = model(**inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "13OSmx5EFOz6"
   },
   "source": [
    "\n",
    "The hidden_rep contains the representation of all the tokens in our sentence and the cls_head contains the representation of the [CLS] token which holds the aggregate representation of the sentence.\n",
    "\n",
    "\n",
    "In this way, we can use the pre-trained M-BERT just like other BERT models. We can use it for fine-tuning the downstream tasks. Now that we have understood how M-BERT works, in the next section, we will evaluate them. "
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "7.01. Understanding multilingual BERT .ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "05c3b23cc9174177bcee786ce119534c": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "084def079e504107ac492665028927f5": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "08ccac7ca81e4a7abfd0c66767590f37": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2e8c53f981b0424f9df0f0e1caa0f73d",
      "placeholder": "​",
      "style": "IPY_MODEL_8ae7e3eb93ab422aa477407a4d3d6705",
      "value": " 714M/714M [00:40&lt;00:00, 17.7MB/s]"
     }
    },
    "0c2d4e7abb974c8dad60d7e1de2eee2e": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_084def079e504107ac492665028927f5",
      "placeholder": "​",
      "style": "IPY_MODEL_8836a23e70504ba79fd2d605e855cf58",
      "value": " 625/625 [00:00&lt;00:00, 15.7kB/s]"
     }
    },
    "1751eccfd3544e6eace561b54ed028a7": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "2d36852fa3c84535b644d8eeae96b117": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_9c93a8681a1b497bb724f7b2e6dadefe",
       "IPY_MODEL_08ccac7ca81e4a7abfd0c66767590f37"
      ],
      "layout": "IPY_MODEL_4d36e204d30c4d0cb8f5d8f4545bf003"
     }
    },
    "2e8c53f981b0424f9df0f0e1caa0f73d": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "30735bf2ffd74418b40f63a8f66c9432": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "32c0276d3e714b0fafa26b8e3e63fb70": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e8f61aa83643415aa917d9ecfbb8d15f",
      "max": 995526,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_1751eccfd3544e6eace561b54ed028a7",
      "value": 995526
     }
    },
    "4b01f8e03f594685a7c24592d9153d63": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4d36e204d30c4d0cb8f5d8f4545bf003": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "536d42bf13324e6b9aa2e93490c259c9": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "700df478ee7147bdb9a511d57aca78a9": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "7d8d6ab425534b4eaab46e866c9c65b7": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8836a23e70504ba79fd2d605e855cf58": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "8ae7e3eb93ab422aa477407a4d3d6705": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "8d1edfd37dcf4a95aecb93046bb5d28b": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4b01f8e03f594685a7c24592d9153d63",
      "placeholder": "​",
      "style": "IPY_MODEL_05c3b23cc9174177bcee786ce119534c",
      "value": " 996k/996k [00:00&lt;00:00, 1.96MB/s]"
     }
    },
    "989710fc8a2c4117b3a49397bdcdc2e5": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "9993840b4fce420499c97f4a6c97d23f": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9fa578e43eb44136879cfd442a506a1c",
      "max": 625,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_700df478ee7147bdb9a511d57aca78a9",
      "value": 625
     }
    },
    "9c93a8681a1b497bb724f7b2e6dadefe": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7d8d6ab425534b4eaab46e866c9c65b7",
      "max": 714314041,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_989710fc8a2c4117b3a49397bdcdc2e5",
      "value": 714314041
     }
    },
    "9fa578e43eb44136879cfd442a506a1c": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "aabc82f34c9948bdb8c3a6fcadbd859b": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_32c0276d3e714b0fafa26b8e3e63fb70",
       "IPY_MODEL_8d1edfd37dcf4a95aecb93046bb5d28b"
      ],
      "layout": "IPY_MODEL_30735bf2ffd74418b40f63a8f66c9432"
     }
    },
    "d9604103dc9d42b39678896f1315125d": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_9993840b4fce420499c97f4a6c97d23f",
       "IPY_MODEL_0c2d4e7abb974c8dad60d7e1de2eee2e"
      ],
      "layout": "IPY_MODEL_536d42bf13324e6b9aa2e93490c259c9"
     }
    },
    "e8f61aa83643415aa917d9ecfbb8d15f": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
